Compiler Flow Logic for Reconfigurable Architectures

ABSTRACT

The technology disclosed partitions a dataflow graph of a high-level program into memory allocations and execution fragments. The memory allocations represent creation of logical memory spaces in on-processor and/or off-processor memories for data required to implement the dataflow graph. The execution fragments represent operations on the data. The technology disclosed designates the memory allocations to virtual memory units and the execution fragments to virtual compute units. The technology disclosed partitions the execution fragments into memory fragments and compute fragments, and assigns the memory fragments to the virtual memory units and the compute fragments to the virtual compute units. The technology disclosed then allocates the virtual memory units to physical memory units and the virtual compute units to physical compute units. It then places the physical memory units and the physical compute units onto positions in the array of configurable units and routes data and control networks between the placed positions.

FIELD OF THE TECHNOLOGY DISCLOSED

The present technology relates to compiler flow of reconfigurablearchitectures, and can be particularly applied to compiler flow logicfor coarse-grain reconfigurable architectures and other distributedexecution systems.

INCORPORATIONS

The following are incorporated by reference for all purposes as if fullyset forth herein:

U.S. Nonprovisional patent application Ser. No. 16/239,252, filed Jan.3, 2019, entitled, “VIRTUALIZATION OF A RECONFIGURABLE DATA PROCESSOR,”(Attorney Docket No. SBNV 1000-1);

U.S. Nonprovisional patent application Ser. No. 16/197,826, filed Nov.21, 2018, entitled, “CONFIGURATION LOAD OF A RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1001-1A);

U.S. Nonprovisional patent application Ser. No. 16/198,086, filed Nov.21, 2018, entitled, “CONFIGURATION UNLOAD OF A RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1001-1B);

U.S. Nonprovisional patent application Ser. No. 16/260,548, filed Jan.29, 2019, entitled, “MATRIX NORMAL/TRANSPOSE READ AND A RECONFIGURABLEDATA PROCESSOR INCLUDING SAME,” (Attorney Docket No. SBNV 1005-1); and

U.S. Nonprovisional patent application Ser. No. 16/407,675, filed May 9,2019, entitled, “CONTROL FLOW BARRIER AND RECONFIGURABLE DATAPROCESSOR,” (Attorney Docket No. SBNV 1007-1).

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Reconfigurable processors, including field programmable gate arrays(FPGAs), can be configured to implement a variety of functions moreefficiently or faster than might be achieved using a general purposeprocessor executing a computer program. So-called coarse-grainreconfigurable architectures (CGRAs) are being developed in which theconfigurable units in the array are more complex than used in typical,more fine-grained FPGAs, and may enable faster or more efficientexecution of various classes of functions. For example, CGRAs have beenproposed that can enable implementation of energy-efficient acceleratorsfor machine learning and artificial intelligence workloads. See,Prabhakar et al., “Plasticine: A Reconfigurable Architecture forParallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada.

CGRAs are an extremely attractive platform when performance, power, orenergy efficiency are paramount. A CGRA is a composition ofcoarse-grained reconfigurable compute and memory elements that areinterconnected together in a certain topology using a reconfigurableinterconnect fabric. It is referred to as coarse-grained reconfigurablebecause the reconfigurable components in the architecture operate at acoarser granularity such as instructions, words, and vectors of words,as opposed to fine-grained, bit-level granularity commonly found inarchitectures such as FPGAs. The programmable data and control paths inCGRAs make them a natural fit to exploit nested parallelism inapplications, by connecting the reconfigurable compute and memorycomponents into customized, deeply nested, and hierarchical pipelines.

Modern applications often have several levels of nested loop levels, andcontain parallelism at multiple levels of nesting. For suchdeeply-nested loops, traditional loop pipelining methods, which focusonly on bodies of the innermost loops, often exploits insufficientparallelism and results poor hardware utilization, resulting in poorperformance, power, or energy efficiency.

Efficient compiler technology enables programmers to describeapplications in a high-level language, while most of the optimizationshappen automatically. Compilers have been proposed that canautomatically translate high-level language to a hierarchy of pipelinesand state machines on FPGAs. See Koeplinger et al., “Spatial: A LanguageAnd Compiler For Application Accelerators,” Proceedings Of The 39th ACMSIGPLAN Conference On Programming Language Design And Implementation(PLDI), 2018.

Achieving the promised performance, power, and energy efficiencycritically hinges on the compiler technology. A CGRA compiler is muchmore complex than a regular compiler because it has to (i) perform codeanalysis to extract task, data, and pipelined parallelism at multiplelevels of nesting, (ii) partition and schedule operations in both spaceand time on the reconfigurable elements, (iii) place the operations ontothe reconfigurable elements, and (iv) route the data and controldependencies between the reconfigurable elements.

Therefore, an opportunity arises to efficiently map nested loops ontothe reconfigurable elements of CGRAs. Improved parallelization andhardware utilization may result.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. The color drawings also may be available in PAIRvia the Supplemental Content tab.

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1 is a system diagram illustrating a system including a compiler, ahost, a memory, and a reconfigurable data processor with an array ofconfigurable units.

FIG. 2 is a block diagram of one implementation of transforming ahigh-level program for mapping onto the reconfigurable data processor.

FIG. 3 is a block diagram of one implementation of allocating thevirtual memory units to multiple physical memory units and allocatingthe virtual compute units to multiple physical compute units.

FIG. 4 is a block diagram of one implementation of fusing the multiplephysical memory units into a single physical memory unit and fusing themultiple physical compute units into a single physical compute unit.

FIG. 5 shows an example of a high-level program for a single layerResNet block.

FIG. 6 depicts one example of the dataflow graph of the ResNet block.

FIGS. 7A, 7B, and 7C illustrate one example of partitioning the dataflowgraph into memory allocations and execution fragments.

FIG. 8 is one implementation of designating the memory allocations tothe virtual memory units and designating the execution fragments to thevirtual compute units.

FIGS. 9A and 9B show one implementation of partitioning the executionfragments into memory fragments and compute fragments.

FIG. 10 depicts one implementation of respectively assigning the memoryfragments to the virtual memory units and respectively assigning thecompute fragments to the virtual compute units.

FIG. 11 illustrates one implementation of mapping the virtual memoryunits to one or more physical memory units and mapping the virtualcompute units to one or more physical compute units.

FIGS. 12A and 12B depict one example of mapping an execution fragment toa virtual compute unit.

FIG. 12C shows that when a single physical compute unit is not enough toexecute an execution fragment mapped to a virtual compute and/or memoryunit, then splitting can be used to map the virtual compute and/ormemory unit to multiple physical compute and/or memory units.

FIG. 13A is one example of fusing physical compute units and physicalmemory units.

FIG. 13B illustrates one example of fusion in space by executingmultiple operations on the single physical compute unit that wouldotherwise execute on separate physical compute units at different clockcycles.

FIG. 13C shows one example of fusion in time by sequentially executingthe multiple operations on the single physical compute unit as separateexecution contexts.

FIG. 14A is a simplified diagram of a tile and an array level networkusable in the reconfigurable data processor of FIG. 1. FIG. 14Billustrates an example switch unit connecting elements in the arraylevel network.

FIG. 15 is a block diagram illustrating an example configurable unit.

DETAILED DESCRIPTION

The following discussion is presented to enable any person skilled inthe art to make and use the technology disclosed, and is provided in thecontext of a particular application and its requirements. Variousmodifications to the disclosed implementations will be readily apparentto those skilled in the art, and the general principles defined hereinmay be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown, but is to be accorded the widest scope consistentwith the principles and features disclosed herein.

Reconfigurable Data Processor & Compiler

FIG. 1 is a system diagram illustrating a system including a compiler196, a host 120, a memory 140, and a reconfigurable data processor 110.As shown in the example of FIG. 1, the reconfigurable data processor 110includes an array 190 of configurable units and a configurationload/unload controller 195. The phrase “configuration load/unloadcontroller”, as used herein, refers to a combination of a configurationload controller and a configuration unload controller. The configurationload controller and the configuration unload controller may beimplemented using separate logic and data path resources, or may beimplemented using shared logic and data path resources as suits aparticular embodiment. In some embodiments, a system may include only aconfiguration load controller of the types described herein. In someembodiments, a system may include only a configuration unload controllerof the types described herein.

Configuration of the array 190 of configurable units involvescompilation of a configuration description by the compiler 196 toproduce a configuration file 198, referred to sometimes as a bitstreamor bit file, and distributing the configuration file 198 to theconfigurable units on the array 190. In one embodiment, the compiler 196provides translations from application programs to bit file.

The processor 110 includes an external I/O interface 130 connected tothe host 120, and external I/O interface 150 connected to the memory140. The I/O interfaces 130, 150 connect via a bus system 115 to thearray 190 of configurable units and to the configuration load/unloadcontroller 195. The bus system 115 may have a bus width of carrying onechunk of data, which can be for this example 128 bits (references to 128bits throughout can be considered as an example chunk size moregenerally). In general, a chunk of the configuration file can have anumber N of bits of data, and the bus system can be configured totransfer N bits of data in one bus cycle, where N is any practical buswidth. A sub-file distributed in the distribution sequence can compriseone chunk, or other amounts of data as suits a particular embodiment.Procedures are described herein using sub-files consisting of one chunkof data each. Of course, the technology can be configured to distributesub-files of different sizes, including sub-files that may comprise twochunks distributed in two bus cycles for example.

To configure configurable units in the array 190 of configurable unitswith a configuration file, the host 120 can send the configuration fileto the memory 140 via the interface 130, the bus system 115, and theinterface 150 in the reconfigurable data processor 110. The host 120connects to the interface 130 via the bus system 125. The memory 140connects to the interface 150 via the bus system 145. The configurationfile can be loaded in many ways, as suits a particular architecture,including in data paths outside the configurable processor 110. Theconfiguration file can be retrieved from the memory 140 via the memoryinterface 150. Chunks of the configuration file can then be sent in adistribution sequence as described herein to configurable units in thearray 190 of configurable units in the reconfigurable data processor110.

An external clock generator 170 or other clock signal sources canprovide a clock signal 175 or clock signals to elements in thereconfigurable data processor 110, including the array 190 ofconfigurable units, and the bus system 115, and the external data I/Ointerfaces.

FIG. 2 is a block diagram 200 of one implementation of transforming ahigh-level program for mapping onto the reconfigurable data processor100. Mapping of an application onto the reconfigurable data processor100 involves mapping of arithmetic and logical operations toreconfigurable units of the reconfigurable data processor 100. Thedesign is specified at a high level of abstraction using machinelearning frameworks like PyTorch, ONNX, and Tensorflow, or high-levellanguages such as C, C++, Java, Python, or Spatial. See, Koeplinger etal., “Spatial: A Language And Compiler For Application Accelerators,”Proceedings Of The 39th ACM SIGPLAN Conference On Programming LanguageDesign And Implementation (PLDI), Proceedings of the 43rd InternationalSymposium on Computer Architecture, 2018. The proposed technique is usedto automatically generate the configuration bits structure thatimplements the behavior described at the high-level of abstraction.

Compiler Flow

High-level program 202 is an application program or source code writtenin programming languages such as (but not restricted to) C, C++, Java,Python, or Spatial. For example, the high-level program 202 canimplement convolutional neural network (CNN) processing with severallayers of varying sizes and data type such that each layer comprisesseveral nested loops with different properties. For example, thehigh-level program 202 can involve memory operations to access theinputs and weights and floating point operations to perform matrixmultiplications. As another example, the high-level program 202 caninclude nested loops with high iteration count and loop bodies that loadand multiply the input values from a preceding layer with the weights ofa succeeding layer to produce the output of the succeeding layer. Thehigh-level program 202 has loop-level parallelism of the outermost loopbody that can be exploited using coarse-grained pipelining. It hasinstruction-level parallelism of the innermost loop body that can besimilarly exploited using loop unrolling, SIMD vectorization, andpipelining.

Regarding loops, loops directly nested in a loop body are termed thechild loops of the outer parent loop. A loop is called an innermost loopif it does not have any children, i.e., there are not any nested loopswithin its body. A loop is an outermost loop if it does not have aparent, i.e., it is not nested within another loop's body. Animperfectly nested loop has a body with a mix of non-looping statements(e.g., primitive arithmetic, logical, and relational operations) and oneor more child loops. Parallelism in the imperfectly nested loops can beexploited at any or all loop levels, and in the operations that compriseloop bodies. Parallelism can occur in multiple forms such asfine-grained and coarse-grained pipeline parallelism, data parallelism,and task parallelism.

At action 1, a dataflow graph generator 204 generates a dataflow graph206 of the high-level program 202. The compilation transforms the inputbehavioral description into an intermediate representation. This firststep may include various code optimizations such as false datadependency elimination, dead-code elimination, and constant folding. Theintermediate representation produced by the compilation exhibits thedata and control dependencies between the operations.

Nodes in the dataflow graph 206 represent control structures, dataoperations, and memory allocations, while edges represent data andeffect dependencies. Each loop in the program is represented as a“controller” in the intermediate representation. The dataflow graph 206supports branches, loops, function calls, and other variations ofcontrol dependencies. Once the dataflow graph 206 is built, additionalanalyses or optimizations can be performed focusing on looptransformations including loop unrolling, loop pipelining, loopfission/fusion, and loop tiling.

At action 2, a partitioner 214, partitions the dataflow graph 206 intomemory allocations 224 and execution fragments 222. Regarding executionfragments 222, they represent operations on the data. An executionfragment can comprise portions of a program representing an amount ofwork. An execution fragment can comprise computations encompassed by aset of loops, a set of graph nodes, or some other unit of work thatrequires synchronization. An execution fragment can comprise a fixed orvariable amount of work, as needed by the program. Similarly, differentones of the execution fragments 222 can contain different amounts ofcomputation. Execution fragments 222 can represent parallel patterns orportions of parallel patterns. Also, the execution fragments 222 areexecutable asynchronously.

In one embodiment, the partitioner 214 comprises a memory allocator 216that generates the memory allocations 224 and an execution fragmentgenerator 212 that generates the execution fragments 222. In oneembodiment, the partitioning of the dataflow graph 206 into theexecution fragments 222 further includes treating calculations within atleast one innermost loop of a nested loop of the dataflow graph 206 as aseparate execution fragment. In another embodiment, the partitioning ofthe dataflow graph 206 into the execution fragments 222 further includestreating calculations of an outer loop around the innermost loop of thedataflow graph 206 as a separate execution fragment. In the case ofimperfectly nested loops, operations within a loop body up to thebeginning of a nested loop within that loop body are grouped together asa separate execution fragment.

Regarding memory allocations 224, they represent the creation of logicalmemory spaces in on-chip and/or off-chip memories for data required toimplement the dataflow graph 206. Memory allocations 224 define the typeand the number of hardware resources (functional units, storage, orconnectivity components). Main memory, abbreviated as DRAM, is anexample of off-chip memory for which the memory allocations 224 aremade. Scratchpad memory, or SRAM, is an example of on-chip memory forwhich the memory allocations 224 are made. Other memory types for whichthe memory allocations 224 can be made for various commonly occurringaccess patterns and layouts found in applications. Examples of suchmemory types include read-only lookup-tables (LUTs), fixed size queues(FIFOs), and register files.

At action 3, a designator 232, designates the memory allocations 224 tovirtual memory units 244 and designates the execution fragments 222 tovirtual compute units 242.

At action 4, an execution fragment partitioner 252, partitions theexecution fragments 222 into memory fragments 264 and compute fragments262. Each memory fragment includes address calculation leading up to amemory access. The compute fragment comprises all other operations inthe parent execution fragment. In one embodiment, each executionfragment is broken up into a plurality of memory fragments and exactlyone compute fragment. The compiler 196 performs the partitioning usingreverse dataflow analysis such that inputs to an address used in amemory access are recursively flagged until the compiler 196 reacheseither constant values or (bound) loop/pattern iterators. A singleexecution fragment may produce one or more memory fragments, dependingon how many memory accesses exist in the original loop body. In caseswhere the same memory addressing logic is shared across multiple memoryaccesses, address calculation may be duplicated to create multiplememory fragments from the same execution fragment.

The memory fragments 264 of the execution fragments 222 are configuredto index into data structures. At least one of the memory fragments 264indexes into a data structure in the logical memory spaces of one of thememory allocations 224. Each compute and memory fragment preservesinformation about all loops whose loop bodies directly contain theoperations in the corresponding execution fragment. In one embodiment,this corresponds to replicating the calculation of the loop iterators ofeach loop into each compute and memory fragment. This replication allowseach fragment to preserve the same iterative behavior as the originalprogram while also allowing distributed calculation of loop iterators.

At action 5, an assigner 266, assigns the memory fragments 264 to thevirtual memory units 244 and assigns the compute fragments 262 to thevirtual compute units 242. The virtual memory units 244 implement thecorresponding memory fragments 264. The virtual compute units 242implement the corresponding compute fragments 262.

Each memory fragment is mapped operation-wise to the virtual memory unitcorresponding to the memory being accessed. Each operation is lowered toits corresponding configuration intermediate representation (IR) forthat virtual memory unit. Each compute fragment is mapped operation-wiseto a newly allocated virtual compute unit. Each operation is lowered toits corresponding configuration intermediate representation (IR) forthat virtual compute unit.

At action 6, an allocator 274, allocates the virtual memory units 244 tophysical memory units 284 and allocates the virtual compute units 242 tophysical compute units 282.

At action 7, a placer and router 286, places the physical memory units284 and the physical compute units 282 onto positions in the array 100of configurable units and routes data and control networks between theplaced positions. In one embodiment, this further includes allocatingphysical resources such as counters and registers within each physicalmemory and compute unit.

At action 8, a bit file generator 292, accesses placement and routinginformation 296 produced by the placer and router 286 and generates thebit file 198 with configuration data for the placed positions and therouted data and control networks. In one embodiment, this includesassigning coordinates and communication resources of the physical memoryand compute units by placing and routing units onto the array 100 ofconfigurable units while maximizing bandwidth and minimizing latency.

At action 9, the configuration load/unload controller 195 loads the bitfile 198 onto an instance of the array 100 of configurable units andcauses the array 100 of configurable units to implement the dataflowgraph 206.

Splitting

FIG. 3 is a block diagram 300 of one implementation of allocating thevirtual memory units 244 to multiple physical memory units 284, 285 andallocating the virtual compute units 242 to multiple physical computeunits 282, 283. This is done at action 6 in FIG. 3 by the allocator 274.In one embodiment, the allocation satisfies hardware constraints of themultiple physical memory units 284, 285 and the multiple physicalcompute units 282, 283.

In one embodiment, the allocating depends, at least in part, on a numberof inputs accepted by a particular physical compute unit. In oneembodiment, as a first step, in each virtual memory and compute unit,operations are removed until the virtual memory and compute unit isphysically realizable. In one embodiment, as a second step, the removedoperations are grouped into a separate, new virtual memory and computeunit. In one embodiment, these two steps are repeated until all virtualmemory and compute units are physically realizable. In one embodiment,the compiler 196 then adds data and control communication channels tothe IR between the virtual memory and compute units based ondependencies in the original virtual memory and compute unit.

In the context of this application, “physically realizable” is modeledusing analysis with target architecture parameters. In one embodiment,the parameters include a capacity of on-chip SRAM available in aphysical memory unit, a number of arithmetic logic unit (ALU) stages, anumber of registers per stage, capabilities of each ALU stage,connections available between ALUs and the registers, and connectionsavailable between the registers. In one embodiment, the order in whichthe operations are removed can vary and is based on heuristics whoseobjective function is to minimize the final number of physicallyrealizable units. In other embodiments, heuristics may be applied with adifferent objective function to minimize the total execution time, whichcould increase the number of physically realizable units.

At action 7, the placer and router 286, places the multiple physicalmemory units 284, 285 and the multiple physical compute units 282, 283onto positions in the array 100 of configurable units and routes dataand control networks between the placed positions.

At action 8, the bit file generator 292, accesses the placement androuting information 296 produced by the placer and router 286 andgenerates the bit file 198 with configuration data for the placedpositions and the routed data and control networks.

At action 9, the configuration load/unload controller 195 loads the bitfile 198 onto an instance of the array 100 of configurable units andcauses the array 100 of configurable units to implement the dataflowgraph 206.

Fusion

FIG. 4 is a block diagram 400 of one implementation of fusing themultiple physical memory units 402, 404, 406 into a single physicalmemory unit 424 and fusing the multiple physical compute units 402, 404,406 into a single physical compute unit 424. This is done at action 7 inFIG. 4 by fuser 424 based on fusion logic 416. The goal of fusion is toreduce resource wastage by better packing operations into physicalmemory and compute units. In one embodiment, termed as “fusion inspace,” two or more physical memory or compute units with underutilizedresources can be combined into a single memory or compute unit with ahigher resource utilization for efficiency, as long as the resultingresource utilization is still physically realizable. In anotherembodiment, termed as “fusion in time,” two or more physical memory orcompute units can be combined by scheduling them to execute sequentiallyas separate execution contexts within a single, new physical memory orcompute unit. In still other embodiments, a combination of fusion rulesin both in space and time may be employed if such a combination ofoptimizations is deemed profitable by the compilation flow. The fusionrules may be determined using heuristics, search algorithms, or otheralgorithmic optimization techniques.

In one embodiment, the fusing depends, at least in part, on a capacityof on-chip SRAM available in a physical memory unit, and a number of ALUstages within the single physical compute unit. In one embodiment, thefusion in space includes executing multiple operations on the singlephysical compute unit 424 that would otherwise execute on separatephysical compute units 402, 404, 406 at different clock cycles. In oneembodiment, the fusion in time includes sequentially executing themultiple operations on the single physical compute unit 424 as separateexecution contexts. In one embodiment, a plurality of operations fromthe dataflow graph 206 grouped onto a particular physical compute unitare mapped onto resources within the particular physical compute unit.In other embodiments, heuristics are used that look for the tradeoffbetween the number of required physical memory and compute units and theachieved performance.

At action 8, the placer and router 286, places the single physicalmemory unit 424 and the single physical compute unit 424 onto positionsin the array 100 of configurable units and routes data and controlnetworks between the placed positions.

At action 9, the bit file generator 292, accesses the placement androuting information 296 produced by the placer and router 286 andgenerates the bit file 198 with configuration data for the placedpositions and the routed data and control networks.

At action 10, the configuration load/unload controller 195 loads the bitfile 198 onto an instance of the array 100 of configurable units andcauses the array 100 of configurable units to implement the dataflowgraph 206.

In some embodiments, the compiler flow logic discussed with reference toFIGS. 2, 3, and 4 is implemented by the compiler 196.

ResNet Example

High-Level Program

FIG. 5 shows an example of the high-level program 202 in PyTorch. Thisexample implements a residual neural network (ResNet) block, which is acommonly used type of Convolutional Neural Network (CNN) popularly usedfor automatic image classification. The ResNet architecture containsseveral layers of convolution operations, where each layer performsseveral convolution operations on the output of the preceding layer andthe weight filters of the current layer. The ResNet architecture alsocontains skip connections that connect outputs of some layers to theinputs of layers that are much further in the network, “skipping” two orthree layers in between. ResNet models also contain nonlinear functionssuch as ReLU and batch normalization in between. Batch normalization isa method for accelerating deep network training by making datastandardization an integral part of the network architecture.

Dataflow Graph

FIG. 6 depicts one example of the dataflow graph 206 of the ResNet block600. This example performs convolution 612 on input 602 and producesoutput 614. Batch normalization 622 is performed on output 614 toproduce output 624. The batch normalized output 624 is then linearizedbetween zero and the maximum positive values by the ReLU activation 632to produce the ReLU activated output 634. Average pooling 642 isperformed on ReLU activated output 634 to produce the average pooledoutput 644. The average pooled output 644 is then fed as input to alinear layer 652 (e.g. a fully-connected network) to produce a finaloutput 654 of the ResNet block 600. The linear layer 652 has 1000neurons (weights).

Dimensionality of the input 602 is 4×3×224×224, where 4 is the batchsize, 3 is the number of input channels (e.g. RGB image channels), 224is the input width (e.g. number of pixel columns in an image), and 224is the input height (e.g. number of pixel widths in an image).Dimensionality of the convolution 612 is 64×3×7×7, where 64 is thenumber of convolution filters, 3 is the number of kernels in eachconvolution filter, 7 is the kernel width (e.g. number of weight columnsin a kernel), and 7 is the kernel height (e.g. number of weight rows ina kernel). Dimensionality of the output 614 is 4×64×112×112, where 4 isthe batch channel, 64 is the number of output channels, 112 is theoutput width (e.g. number of feature columns in a feature map), and 112is the output height (e.g. number of feature rows in a feature map).Dimensionality of the final output 654 is 4×100, where 4 is the batchchannel and 1000 is the number of output channels.

Intermediate Representation & Partitioning

FIGS. 7A, 7B, 7C show example implementations of a subset of the ResNetarchitecture as the high-level program 202, which is representedinternally as the dataflow graph 206. In one embodiment, the high-levelprogram could be written in the Spatial high-level language. See, D.Koeplinger et al., “Spatial: A Language And Compiler For ApplicationAccelerators,” Proceedings Of The 39th ACM SIGPLAN Conference OnProgramming Language Design And Implementation (PLDI), Proceedings ofthe 43rd International Symposium on Computer Architecture, 2018.

FIGS. 7A, 7B, and 7C illustrate one example of partitioning the dataflowgraph 206 into memory allocations 224 and execution fragments 222. InFIGS. 7A, 7B, and 7C, blue highlighted code identifies the memoryallocations 224 and green highlighted code identifies the executionfragments 222. The first memory allocation 701 allocates memory spacesin on-chip SRAM for the input 602. The second memory allocation 702allocates memory spaces in the on-chip SRAM for the convolution 612. Thethird memory allocation 703 allocates memory spaces in the on-chip SRAMfor the output 614.

The first execution fragment 704 implements the convolution 612 betweenthe input 602 and convolution weights of the convolution 612. The secondexecution fragment 705 implements accumulation of the output 614.

The fourth memory allocation 706 allocates memory spaces in the on-chipSRAM for the output 624 of the batch normalization 622. The fifth memoryallocation 707 allocates memory spaces in the on-chip SRAM for thescaling value of the batch normalization 622. The sixth memoryallocation 708 allocates memory spaces in the on-chip SRAM for the biasvalue of the batch normalization 622.

The third execution fragment 709 implements mean calculation 1302 of thebatch normalization 622. The fourth execution fragment 710 implementsmean normalization 1206 of the batch normalization 622.

In FIG. 7B, the fifth execution fragment 711 implements variancecalculation 1304 of the batch normalization 622. The sixth executionfragment 712 implements variance normalization 1207 of the batchnormalization 622. The seventh execution fragment 713 implementscalculation of the output 624 of the batch normalization 622.

The seventh memory allocation 714 allocates memory spaces in the on-chipSRAM for the output 634 of the ReLU activation 632. The eighth executionfragment 715 implements calculation of the output 634 of the ReLUactivation 632.

The eighth memory allocation 716 allocates memory spaces in the on-chipSRAM for the output 644 of the average pooling 642. The ninth executionfragment 717 implements calculation of the output 644 of the averagepooling 642. The tenth execution fragment 718 implements accumulation ofthe output 644 of the average pooling 642.

In FIG. 7C, the ninth memory allocation 719 allocates memory spaces inthe on-chip SRAM for the neurons (weights) of the linear layer 652. Thetenth memory allocation 720 allocates memory spaces in the on-chip SRAMfor the final output 654. The eleventh execution fragment 721 implementscalculation of the final output 654.

Designation

FIG. 8 shows an example of one implementation of designating the memoryallocations 224 to the virtual memory units 244 and designating theexecution fragments to the virtual compute units 242. In FIG. 8, the tenmemory allocations 224 in FIGS. 7A, 7B, and 7C are respectivelydesignated a corresponding virtual memory unit (VMU). Also in FIG. 8,the eleven execution fragments in FIGS. 7A, 7B, and 7C are respectivelydesignated a corresponding virtual compute unit (VCU).

Execution Fragment Partitioning

FIGS. 9A and 9B show one implementation of partitioning the executionfragments 222 into memory fragments 264 and compute fragments 262. FIG.9A shows that the first execution fragment 704, which implements theconvolution 612 between the input 602 and convolution weights of theconvolution 612, is partitioned into a first memory fragment 901, asecond memory fragment 902, and a compute fragment 903. FIG. 9B alsoshows the respective addresses computed by the first and second memoryfragments 901, 902 (MF1 a, MF1 b) for memory access.

FIG. 10 depicts one implementation of respectively assigning the memoryfragments 901, 902 to the virtual memory units 1002, 1022 (VMU 1, VMU 2)and assigning the compute fragment 903 to the virtual compute unit 1014(VCU 1). The address calculation 901, 902 (MF1 a, MF1 b) arerespectively allocated to the virtual memory units 1002, 1022 (VMU 1,VMU 2) with the allocations 904, 905 (A1, A2).

FIG. 11 illustrates one implementation of mapping the virtual memoryunits 1002, 1022 (VMU 1, VMU 2) to one or more physical memory units1102, 1112, 1122, 1132, 1142 (PMU 1_0, PMU 1_1, PMU 2_0, PMU 2_1, PMU2_2) and mapping the virtual compute unit 1014 (VCU 1) to one or morephysical compute units 1104, 1124 (PCU 1_0, PCU 1_1).

The virtual memory unit 1002 (VMU 1) is mapped to the physical memoryunit 1102 (PMU 1_0) and the physical memory unit 1112 (PMU 1_1). Theoriginal address designation 904 (A1) of the virtual memory unit 1002(VMU 1) is expressed as duplicated address designations 1101, 1102(A1_0, A1_1) of the physical memory units 1102, 1112 (PMU 1_0, PMU 1_1),respectively.

The virtual memory unit 1022 (VMU 2) is mapped to the physical memoryunit 1122 (PMU 2_0), the physical memory unit 1132 (PMU 2_1), and thephysical memory unit 1142 (PMU 2_1). The original address designation905 (A2) of the virtual memory unit 1022 (VMU 2) is expressed asduplicated address designations 1103, 1104, 1105 (A2_0, A2_1, A2_2) ofthe physical memory units 1122, 1132, 1142 (PMU 2_0, PMU 2_1, PMU 2_2),respectively.

The virtual compute unit 1014 (VCU 1) is mapped to the physical computeunit 1104 (PCU 1_0) and the physical compute unit 1124 (PCU 1_1).

FIGS. 12A and 12B depict one example of mapping execution fragment 713to a virtual compute unit (VCU 7) that has four inputs. Executionfragment 713 is partitioned into a first memory fragment 1201, a secondmemory fragment 1202, a third memory fragment 1203, a fourth memoryfragment 1205, and a compute fragment 1204.

In FIG. 12B, since the virtual compute unit (VCU 7) has four inputs, itis able to process all the four memory fragments 1201, 1202, 1203, 1205to compute the compute fragment 1204 for the execution fragment 713.

When a single physical compute unit is not enough to execute anexecution fragment mapped to a virtual compute and/or memory unit, thensplitting can be used to map the virtual compute and/or memory unit tomultiple physical compute and/or memory units. The multiple physicalcompute and/or memory units then together execute the executionfragment.

Turning to FIG. 12C, consider that a single physical compute unit hasonly three inputs. However though, the virtual compute unit (VCU 7) hasfour inputs. To account for this, two physical compute units (PCU 7_0and PCU 7_1) are used to compute the compute fragment 1204 for theexecution fragment 713. This is achieved by mapping the virtual computeunit (VCU 7) to the two physical compute units (PCU 7_0 and PCU 7_1).

FIG. 13A is one example of fusing physical compute units and physicalmemory units. In FIG. 13A, physical compute units and physical memoryunits implementing various memory fragments and compute fragments areconnected. The connections follow data and control dependencies requiredby the dataflow graph 206.

Fusion in Space

FIG. 13B illustrates one example of fusion in space by executingmultiple operations on the single physical compute unit that wouldotherwise execute on separate physical compute units at different clockcycles. In FIG. 13B, PCU 3_0 and PCU 4_0 are combined into a single PCU3_0_4_0. Also, PCU 5_0 and PCU 6_0 are combined into a single PCU5_0_6_0.

Fusion in Time

FIG. 13C shows one example of fusion in time by sequentially executingthe multiple operations on the single physical compute unit as separateexecution contexts. In FIG. 13C, PCU 3_0, PCU 4_0, PCU 5_0, and PCU 6_0are combined into a single PCU 3_0_4_0_5_0_6_0. This means the singlePCU 3_0_4_0_5_0_6_0 executes the fused compute fragments 1302, 1206 (CF3/4) for one segment of time, then changes contexts and executes thefused compute fragments 1304, 1207 (CF 5/6). The value after CF 3/4 ispassed back to the unit (since it is an intermediate value used as aninput to CF 5/6). The implementation of this self-loop depends on thehardware embodiment, but can be implemented as an internal register ormemory in the PCU or using the external network by feeding the PCU'soutput back into itself. In another embodiment, the data fed back to thePCU can be stored in a separate memory unit such as a PMU or a set ofPMUs.

Reconfigurable Tile

FIG. 14A is a simplified diagram 1400 of a tile and an array levelnetwork usable in the reconfigurable data processor of FIG. 1. FIG. 14Billustrates an example switch unit connecting elements in the arraylevel network. In this example, the array of configurable units 300includes a plurality of types of configurable units. The types ofconfigurable units in this example, include Pattern Compute Units (PCU),Pattern Memory Units (PMU), switch units (S), and Address Generation andCoalescing Units (each including two address generators AG and a sharedCU). For an example of the functions of these types of configurableunits, see, Prabhakar et al., “Plasticine: A Reconfigurable ArchitectureFor Parallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON, Canada,which is incorporated by reference as if fully set forth herein.

Each of these configurable units contains a configuration storecomprising a set of registers or flip-flops that represent either thesetup or the sequence to run a program, and can include the number ofnested loops, the limits of each loop iterator, the instructions to beexecuted for each stage, the source of the operands, and the networkparameters for the input and output interfaces.

Additionally, each of these configurable units contains a configurationstore comprising a set of registers or flip-flops that store statususable to track progress in nested loops or otherwise. The configurationfile 198 contains a bitstream representing the initial configuration, orstarting state, of each of the components that execute the program. Thisbitstream is referred to as a bit file. Program load is the process ofsetting up the configuration stores in the array 190 of configurableunits based on the contents of the bit file to allow all the componentsto execute a program (i.e., a machine). Program Load may also requirethe load of all PMU memories.

The array level network includes links interconnecting configurableunits in the array. The links in the array level network include one ormore and, in this case three, kinds of physical buses: a chunk-levelvector bus (e.g. 128 bits of data), a word-level scalar bus (e.g. 32bits of data), and a multiple bit-level control bus. For instance,interconnect 1421 between switch units 1411 and 1412 includes a vectorbus interconnect with vector bus width of 128 bits, a scalar businterconnect with a scalar bus width of 32 bits, and a control businterconnect.

The three kinds of physical buses differ in the granularity of databeing transferred. In one embodiment, the vector bus can carry a chunkthat includes 16-Bytes (=128 bits) of data as its payload. The scalarbus can have a 32-bit payload, and carry scalar operands or controlinformation. The control bus can carry control handshakes such as tokensand other signals. The vector and scalar buses can be packet switched,including headers that indicate a destination of each packet and otherinformation such as sequence numbers that can be used to reassemble afile when the packets are received out of order. Each packet header cancontain a destination identifier that identifies the geographicalcoordinates of the destination switch unit (e.g. the row and column inthe array), and an interface identifier that identifies the interface onthe destination switch (e.g. North, South, East, West, etc.) used toreach the destination unit. The control network can be circuit switchedbased on timing circuits in the device, for example. The configurationload/unload controller can generate a header for each chunk ofconfiguration data of 128 bits. The header is transmitted on a headerbus to each configurable unit in the array 190 of configurable units.

In one example, a chunk of data of 128 bits is transmitted on the vectorbus that provides the chunk as vector inputs to a configurable unit. Thevector bus can include 128 payload lines, and a set of header lines. Theheader can include a sequence ID for each chunk, which can includes:

A bit to indicates if the chunk is scratchpad memory or configurationstore data.

Bits that form a chunk number.

Bits that indicate a column identifier.

Bits that indicate a row identifier.

Bits that indicate a component identifier.

For a load operation, the configuration load controller can send thenumber N of chunks to a configurable unit in order from N−1 to 0. Forthis example, the 6 chunks are sent out in most significant bit firstorder of Chunk 5->Chunk 4->Chunk 3->Chunk 2->Chunk 1->Chunk 0. (Notethat this most significant bit first order results in Chunk 5 beingdistributed in round 0 of the distribution sequence from the arrayconfiguration load controller.) For an unload operation, theconfiguration unload controller can write out the unload data of orderto the memory. For both load and unload operations, the shifting in theconfiguration serial chains in a configuration data store in aconfigurable unit is from LSB (least-significant-bit) to MSB(most-significant-bit), or MSB out first.

FIG. 14B illustrates an example switch unit connecting elements in thearray level network. As shown in the example of FIG. 14B, a switch unitcan have 8 interfaces. The North, South, East and West interfaces of aswitch unit are used for connections between switch units. TheNortheast, Southeast, Northwest and Southwest interfaces of a switchunit are each used to make connections to PCU or PMU instances. A set of2 switch units in each tile quadrant have connections to an AddressGeneration and Coalescing Unit (AGCU) that include multiple addressgeneration (AG) units and a coalescing unit (CU) connected to themultiple address generation units. The coalescing unit (CU) arbitratesbetween the AGs and processes memory requests. Each of the 8 interfacesof a switch unit can include a vector interface, a scalar interface, anda control interface to communicate with the vector network, the scalarnetwork, and the control network.

During execution of a machine after configuration, data can be sent viaone or more unit switches and one or more links between the unitswitches to the configurable units using the vector bus and vectorinterface(s) of the one or more switch units on the array level network.

In embodiments described herein, a configuration file or bit file 198,before configuration of the tile, can be sent from the configurationload controller using the same vector bus, via one or more unit switchesand one or more links between the unit switches to the configurable unitusing the vector bus and vector interface(s) of the one or more switchunits on the array level network. For instance, a chunk of configurationdata in a unit file particular to a configurable unit PMU 1441 can besent from the configuration load/unload controller 1401 to the PMU 1441,via a link 1420 between the configuration load/unload controller 1401and the West (W) vector interface of the switch unit 1411, the switchunit 1411, and a link 1431 between the Southeast (SE) vector interfaceof the switch unit 1411 and the PMU 1441.

In this example, one of the AGCUs is configured to be a master AGCU,which includes a configuration load/unload controller (e.g. 1401). Themaster AGCU implements a register through which the host (120, FIG. 1)can send commands via the bus system to the master AGCU. The master AGCUcontrols operations on an array of configurable units in a tile andimplements a program control state machine to track the state of thetile based on the commands it receives from the host through writes tothe register. For every state transition, the master AGCU issuescommands to all components on the tile over a daisy chained command bus(FIG. 15). The commands include a program reset command to resetconfigurable units in an array of configurable units in a tile, and aprogram load command to load a configuration file to the configurableunits.

The configuration load controller in the master AGCU is responsible forreading the configuration file from the memory and sending theconfiguration data to every configurable unit of the tile. The masterAGCU can read the configuration file from the memory at preferably themaximum throughput of the top level network. The data read from memoryare transmitted by the master AGCU over the vector interface on thearray level network to the corresponding configurable unit according toa distribution sequence described herein.

In one embodiment, in a way that can reduce the wiring requirementswithin a configurable unit, configuration and status registers holdingunit files to be loaded in a configuration load process, or unloaded ina configuration unload process in a component are connected in a serialchain and can be loaded through a process of shifting bits through theserial chain. In some embodiments, there may be more than one serialchain arranged in parallel or in series. When a configurable unitreceives the for example 128 bits of configuration data from the masterAGCU in one bus cycle, the configurable unit shifts this data throughits serial chain at the rate of 1 bit per cycle, where shifter cyclescan run at the same rate as the bus cycle. It will take 128 shiftercycles for a configurable unit to load 128 configuration bits with the128 bits of data received over the vector interface. The 128 bits ofconfiguration data are referred to as a chunk. A configurable unit canrequire multiple chunks of data to load all its configuration bits.

The configurable units interface with the memory through multiple memoryinterfaces (150, FIG. 1). Each of the memory interfaces can be accessedusing several AGCUs. Each AGCU contains a reconfigurable datapath togenerate requests for the off-chip memory. Each AGCU contains FIFOs(first-in-first-out buffers for organizing data) to buffer outgoingcommands, data, and incoming responses from the off-chip memory.

The address generators AGs in the AGCUs can generate memory commandsthat are either dense or sparse. Dense requests can be used to bulktransfer contiguous off-chip memory regions, and can be used to read orwrite chunks of data from/to configurable units in the array ofconfigurable units. Dense requests can be converted to multiple off-chipmemory burst requests by the coalescing unit (CU) in the AGCUs. Sparserequests can enqueue a stream of addresses into the coalescing unit. Thecoalescing unit can use a coalescing cache to maintain metadata onissued off-chip memory requests to combine sparse addresses that belongto the same off-chip memory request to minimize the number of issuedoff-chip memory requests.

Reconfigurable Units

FIG. 15 is a block diagram illustrating an example configurable unit1500, such as a Pattern Compute Unit (PCU). In the context of thisapplication, a PCU corresponds to a physical compute unit. Configurableunits in the array of configurable units include configuration datastores 1520 (e.g. serial chains) to store unit files comprising aplurality of chunks (or sub-files of other sizes) of configuration dataparticular to the corresponding configurable units. Configurable unitsin the array of configurable units each include unit configuration loadlogic 1540 connected to the configuration data store 1520 via line 1522,to execute a unit configuration load process. The unit configurationload process includes, receiving via the bus system (e.g. the vectorinputs), chunks of a unit file particular to the configurable unit, andloading the received chunks into the configuration data store 1520 ofthe configurable unit.

The configuration data stores in configurable units in the plurality ofconfigurable units in this example comprise serial chains of latches,where the latches store bits that control configuration of the resourcesin the configurable unit. A serial chain in a configuration data storecan include a shift register chain for configuration data and a secondshift register chain for state information and counter values connectedin series.

A configurable unit can interface with the scalar, vector, and controlbuses using three corresponding sets of inputs and outputs (IO): scalarinputs/outputs, vector inputs/outputs, and control inputs/outputs.Scalar IOs can be used to communicate single words of data (e.g. 32bits). Vector IOs can be used to communicate chunks of data (e.g. 128bits), in cases such as receiving configuration data in a unitconfiguration load process, and transmitting and receiving data duringoperation after configuration across a long pipeline between multiplePCUs. Control IOs can be used to communicate control signals such as thestart or end of execution of a configurable unit. Control inputs arereceived by control block 1570, and control outputs are provided by thecontrol block 1570.

Each vector input is buffered using a vector FIFO in a vector FIFO block1560 which can include one or more vector FIFOs. Each scalar input isbuffered using a scalar FIFO 1550. Using input FIFOs decouples timingbetween data producers and consumers, and simplifiesinter-configurable-unit control logic by making it robust to input delaymismatches.

Input configuration data 1510 can be provided to a vector FIFO as vectorinputs, and then be transferred to the configuration data store 1520.Output configuration data 1530 can be unloaded from the configurationdata store 1520 using the vector outputs.

The CGRA uses a daisy chained completion bus to indicate when aload/unload command has been completed. The master AGCU transmits theprogram load and unload commands to configurable units in the array ofconfigurable units over a daisy-chained command bus. As shown in theexample of FIG. 15, a daisy chained completion bus 1591 and a daisychained command bus 1592 are connected to daisy chain logic 1593, whichcommunicates with the unit configuration load logic 1540. The daisychain logic 1593 can include load complete status logic, as describedbelow. The daisy chained completion bus is further described below.Other topologies for the command and completion buses are clearlypossible but not described here.

A configurable unit includes multiple reconfigurable datapaths in block1580. A datapath in a configurable unit can be organized as amulti-stage (Stage 1 . . . Stage N), reconfigurable SIMD (SingleInstruction, Multiple Data) pipeline. The chunks of data pushed into theconfiguration serial chain in a configurable unit include configurationdata for each stage of each datapath in the configurable unit. Theconfiguration serial chain in the configuration data store 420 isconnected to the multiple datapaths in block 1580 via lines 1521.

In the context of this application, a pattern memory unit (PMU)corresponds to a physical memory unit. A PMU can contain scratchpadmemory coupled with a reconfigurable datapath intended for addresscalculation, along with the bus interfaces used in the PCU. PMUs can beused to distribute on-chip memory throughout the array of reconfigurableunits. In one embodiment, address calculation within the memory in thePMUs is performed on the PMU datapath, while the core computation isperformed within the PCU. Each PMU contains a programmer-managedscratchpad memory coupled with a reconfigurable datapath intendedprimarily for address calculation, and other compute operations asrequired by the program. PMUs are used to distribute on-chip memorythroughout the array 190. The array architecture makes a distinctionbetween the operations involved in memory addresses calculation and thecore computation underlying applications. Address calculation isperformed on the PMU datapath, while the core computation is performedwithin the PCU. Several observations have motivated this design choice:(i) address calculation involves simple scalar math, which requiressimpler ALUs than the ALUs in PCUs; (ii) Using multiple lanes foraddress computation is often unnecessary for most on-chip accesspatterns; and (iii) Performing address calculation within the PCUrequires routing the addresses from the PCU to the PMU, which occupiesPCU stages and output links, and can lead to PCU under-utilization.

PCUs and PMUs (collectively “units”) communicate with three kinds ofinterconnect: word-level scalar, multiple-word-level vector, andbit-level control interconnects. The array 190 of configurable unitsinterfaces with DRAM through multiple DDR channels. Each channel has anassociated address management unit that arbitrates between multipleaddress streams, and consists of buffers to support multiple outstandingmemory requests and address coalescing to minimize DRAM accesses. Localaddress calculation is done in PMUs, DRAM address computation happens inthe DRAM address management units, and the remaining data computationhappens in PCUs. The scratchpads are built with multiple SRAM banksmatching the number of PCU lanes. Address decoding logic around thescratchpad can be configured to operate in several banking modes tosupport various access patterns. Strided banking mode supports linearaccess patterns often found on dense data structures. FIFO mode supportsstreaming accesses. Line buffer mode captures access patterns resemblinga sliding window. Duplication mode, where the contents are duplicatedacross all memory banks, provides multiple read address channels tosupport parallelized on-chip gather operations.

The PCU is designed to execute innermost parallel patterns in anapplication. The PCU datapath is organized as a multi-stage,reconfigurable SIMD pipeline. This design enables each PCU to achievehigh compute density, and exploit both loop-level parallelism acrosslanes and pipeline parallelism across stages. Each stage of each SIMDlane is composed of a functional unit (FU) and associated pipelineregisters (PR). FUs perform 32 bit word-level arithmetic and binaryoperations, including support for floating point and integer operations.As the FUs in a single pipeline stage operate in SIMD, each stagerequires only a single configuration register. Results from each FU arewritten to its associated register. PRs in each lane are chainedtogether across pipeline stages to allow live values propagate betweenstages within the same lane. Cross-lane communication between FUs iscaptured using two types of intra-PCU networks: a reduction tree networkthat allows reducing values from multiple lanes into a single scalar,and a shift network which allows using PRs as sliding windows acrossstages to exploit reuse in stencil applications. Both networks usededicated registers within PRs to minimize hardware overhead.

PCUs interface with the global interconnect using three kinds of inputsand outputs (IO); scalar, vector, and control. Scalar IO is used tocommunicate single words of data, such as the results of Folds. Eachvector IO allows communicating one word per lane in the PCU, and is usedin cases such as reading and writing to scratchpads in PMUs andtransmitting intermediate data across a long pipeline between multiplePCUs. Each vector and scalar input is buffered using a small FIFO. Usinginput FIFOs decouples data producers and consumers, and simplifiesinter-PCU control logic by making it robust to input delay mismatches.Control IO is used to communicate control signals such as the start orend of execution of a PCU, or to indicate backpressure.

A reconfigurable chain of counters generates pattern iteration indicesand control signals to coordinate execution. PCU execution begins whenthe control block enables one of the counters. Based on theapplication's control and data dependencies, the control block can beconfigured to combine multiple control signals from both local FIFOs andglobal control inputs to trigger PCU execution. The control block isimplemented using reconfigurable combinational logic and programmableup-down counters for state machines.

Just as banking is important to feed multiple SIMD units to sustaincompute throughput, N-buffering, or generalized double buffering, isjust as important to support coarse-grained pipelines. As an example,the skip connections in ResNet, and the buffers that hold the outputs ofeach layer can be implemented using N-buffering. The PMU scratchpad canbe configured to operate as an N-buffer with any of the banking modesdescribed. N-buffers are implemented by partitioning the address spacein each SRAM bank into N disjoint regions. Using write and read stateinformation, an appropriate offset is added to each bank's local addressto access the correct data.

A programmable counter chain and control block triggers PMU executionsimilar to the PCU. Each PMU typically contains write addresscalculation logic from the producer pattern, and read addresscalculation logic from the consumer pattern. Based on the state of thelocal FIFOs and external control inputs, the control block can beconfigured to trigger the write address computation, read addresscomputation, or both, by enabling the appropriate counters.

Particular Implementations

In one implementation, we disclose a computer-implemented method oftransforming a high-level program for mapping onto a reconfigurable dataprocessor with an array of configurable units. The method includespartitioning a dataflow graph of the high-level program into memoryallocations and execution fragments. The memory allocations representcreation of logical memory spaces in on-processor and/or off-processormemories for data required to implement the dataflow graph. Theexecution fragments represent operations on the data.

The method includes designating the memory allocations to virtual memoryunits and designating the execution fragments to virtual compute units.

The method includes partitioning the execution fragments into memoryfragments and compute fragments.

The method includes assigning the memory fragments to the virtual memoryunits and assigning the compute fragments to the virtual compute units.

The method includes allocating the virtual memory units to physicalmemory units and allocating the virtual compute units to physicalcompute units.

The method includes placing the physical memory units and the physicalcompute units onto positions in the array of configurable units androuting data and control networks between the placed positions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

Each of the features discussed in the particular implementation sectionfor other implementations apply equally to this implementation. Asindicated above, all the other features are not repeated here and shouldbe considered repeated by reference. The reader will understand howfeatures identified in these implementations can readily be combinedwith sets of base features identified in other implementations.

The method includes allocating at least one of the virtual memory unitsto multiple physical memory units and/or allocating at least one of thevirtual compute units to multiple physical compute units.

The method includes placing the multiple physical memory units and themultiple physical compute units onto positions in the array ofconfigurable units and routing data and control networks between theplaced positions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

The allocating satisfies hardware constraints of the multiple physicalmemory units and the multiple physical compute units. The allocatingdepends, at least in part, on a number of inputs accepted by aparticular physical compute unit. The allocating depends, at least inpart, on a capacity of on-chip SRAM available in a physical memory unit,a number of arithmetic logic unit (ALU) stages, a number of registersper stage, capabilities of each ALU stage, connections available betweenALUs and the registers, and connections available between the registers.

The method includes fusing at least two of the physical memory unitsinto a single physical memory unit and/or fusing at least two of thephysical compute units into a single physical compute unit.

The method includes placing the single physical memory unit and thesingle physical compute unit onto positions in the array of configurableunits and routing data and control networks between the placedpositions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

The fusion increases utilization of resources of the single physicalmemory unit and the single physical compute unit. The fusing depends, atleast in part, on a number of ALU stages within the single physicalcompute unit. The fusion in space includes executing multiple operationson the single physical compute unit that would otherwise execute onseparate physical compute units at different clock cycles. The fusion intime includes sequentially executing the multiple operations on thesingle physical compute unit as separate execution contexts. Theexecution fragments are executable asynchronously.

The method includes configuring the memory fragments of the executionfragments to index into data structures. At least one of the memoryfragments indexes into a data structure in the logical memory spaces ofone of the memory allocations. At least one of the execution fragmentsincludes a loop or pattern iterator that causes operations to at leastone of the compute fragments to operate repeatedly over values indexedby the memory fragment. The compute fragment and the memory fragment ofa particular execution fragment cause the operations to operate over atleast two dimensions of a data structure holding the values indexed. Aparticular execution fragment includes multiple memory fragments thatindex into multiple data structures.

The partitioning into the execution fragments further includes treatingcalculations within at least one inner-most loop of a nested loop of thedataflow graph as a separate execution fragment. The partitioning intothe execution fragments further includes treating calculations of anouter loop around the inner-most loop of the dataflow graph as aseparate execution fragment.

The method includes mapping a plurality of operations from the dataflowgraph that are grouped onto a particular physical compute unit ontoresources within the particular physical compute unit.

Other implementations of the method described in this section caninclude a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation of the method described inthis section can include a system including memory and one or moreprocessors operable to execute instructions, stored in the memory, toperform any of the methods described above.

In one implementation, we disclose a computer-implemented method oftransforming a high-level program for mapping onto a reconfigurable dataprocessor with an array of configurable units.

The method includes partitioning a dataflow graph of the high-levelprogram into memory allocations and execution fragments. The memoryallocations represent creation of logical memory spaces in on-processorand/or off-processor memories for data required to implement thedataflow graph. The execution fragments represent operations on thedata.

The method includes designating the memory allocations and the executionfragments to virtual configurable units.

The method includes partitioning the execution fragments into memoryfragments and compute fragments.

The method includes assigning the memory fragments and the computefragments to the virtual configurable units.

The method includes allocating the virtual configurable units tophysical configurable units.

The method includes placing the physical configurable units ontopositions in the array of configurable units and routing data andcontrol networks between the placed positions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

Each of the features discussed in the particular implementation sectionfor other implementations apply equally to this implementation. Asindicated above, all the other features are not repeated here and shouldbe considered repeated by reference. The reader will understand howfeatures identified in these implementations can readily be combinedwith sets of base features identified in other implementations.

The method includes allocating at least one of the virtual configurableunits to multiple physical configurable units.

The method includes placing the multiple physical configurable unitsonto positions in the array of configurable units and routing data andcontrol networks between the placed positions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

The method includes fusing at least two of the physical configurableunits into a single physical configurable unit.

The method includes placing the single physical configurable unit ontopositions in the array of configurable units and routing data andcontrol networks between the placed positions.

The method includes generating a bit file with configuration data forthe placed positions and the routed data and control networks. The bitfile, when loaded onto an instance of the array of configurable units,causes the array of configurable units to implement the dataflow graph.

Other implementations of the method described in this section caninclude a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation of the method described inthis section can include a system including memory and one or moreprocessors operable to execute instructions, stored in the memory, toperform any of the methods described above.

The preceding description is presented to enable the making and use ofthe technology disclosed. Various modifications to the disclosedimplementations will be apparent, and the general principles definedherein may be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown, but is to be accorded the widest scope consistentwith the principles and features disclosed herein. The scope of thetechnology disclosed is defined by the appended claims.

What is claimed is:
 1. A computer-implemented method of transforming ahigh-level program for mapping onto a reconfigurable data processor withan array of configurable units, the method including: partitioning adataflow graph of the high-level program into memory allocations andexecution fragments, wherein the memory allocations represent creationof logical memory spaces in on-processor and/or off-processor memoriesfor data required to implement the dataflow graph, and the executionfragments represent operations on the data; designating the memoryallocations to virtual memory units and designating the executionfragments to virtual compute units; partitioning the execution fragmentsinto memory fragments and compute fragments; assigning the memoryfragments to the virtual memory units and assigning the computefragments to the virtual compute units; allocating the virtual memoryunits to physical memory units and allocating the virtual compute unitsto physical compute units; placing the physical memory units and thephysical compute units onto positions in the array of configurable unitsand routing data and control networks between the placed positions; andgenerating a bit file with configuration data for the placed positionsand the routed data and control networks, wherein the bit file, whenloaded onto an instance of the array of configurable units, causes thearray of configurable units to implement the dataflow graph.
 2. Thecomputer-implemented method of claim 1, further including: allocating atleast one of the virtual memory units to multiple physical memory unitsand/or allocating at least one of the virtual compute units to multiplephysical compute units; placing the multiple physical memory units andthe multiple physical compute units onto positions in the array ofconfigurable units and routing data and control networks between theplaced positions; and generating a bit file with configuration data forthe placed positions and the routed data and control networks, whereinthe bit file, when loaded onto an instance of the array of configurableunits, causes the array of configurable units to implement the dataflowgraph.
 3. The computer-implemented method of claim 2, wherein theallocating satisfies hardware constraints of the multiple physicalmemory units and the multiple physical compute units.
 4. Thecomputer-implemented method of claim 2, wherein the allocating depends,at least in part, on a number of inputs accepted by a particularphysical compute unit.
 5. The computer-implemented method of claim 2,wherein the allocating depends, at least in part, on a capacity ofon-chip SRAM available in a physical memory unit, a number of arithmeticlogic unit (ALU) stages, a number of registers per stage, capabilitiesof each ALU stage, connections available between ALUs and the registers,and connections available between the registers.
 6. Thecomputer-implemented method of claim 1, further including: fusing atleast two of the physical memory units into a single physical memoryunit and/or fusing at least two of the physical compute units into asingle physical compute unit; placing the single physical memory unitand the single physical compute unit onto positions in the array ofconfigurable units and routing data and control networks between theplaced positions; and generating a bit file with configuration data forthe placed positions and the routed data and control networks, whereinthe bit file, when loaded onto an instance of the array of configurableunits, causes the array of configurable units to implement the dataflowgraph.
 7. The computer-implemented method of claim 6, wherein the fusionincreases utilization of resources of the single physical memory unitand the single physical compute unit.
 8. The computer-implemented methodof claim 6, wherein the fusing depends, at least in part, on a capacityof on-chip SRAM available in a physical memory unit, and a number of ALUstages within the single physical compute unit.
 9. Thecomputer-implemented method of claim 6, wherein the fusion in spaceincludes executing multiple operations on the single physical computeunit that would otherwise execute on separate physical compute units atdifferent clock cycles.
 10. The computer-implemented method of claim 9,wherein the fusion in time includes sequentially executing the multipleoperations on the single physical compute unit as separate executioncontexts.
 11. The computer-implemented method of claim 1, wherein theexecution fragments are executable asynchronously.
 12. Thecomputer-implemented method of claim 11, further including: configuringthe memory fragments of the execution fragments to index into datastructures.
 13. The computer-implemented method of claim 12, wherein atleast one of the memory fragments indexes into a data structure in thelogical memory spaces of one of the memory allocations.
 14. Thecomputer-implemented method of claim 12, wherein at least one of theexecution fragments includes a loop or pattern iterator that causesoperations to at least one of the compute fragments to operaterepeatedly over values indexed by the memory fragment.
 15. Thecomputer-implemented method of claim 14, wherein the compute fragmentand the memory fragment of a particular execution fragment cause theoperations to operate over at least two dimensions of a data structureholding the values indexed.
 16. The computer-implemented method of claim12, wherein a particular execution fragment includes multiple memoryfragments that index into multiple data structures.
 17. Thecomputer-implemented method of claim 1, wherein the partitioning intothe execution fragments further includes treating calculations within atleast one inner-most loop of a nested loop of the dataflow graph as aseparate execution fragment.
 18. The computer-implemented method ofclaim 17, wherein the partitioning into the execution fragments furtherincludes treating calculations of an outer loop around the inner-mostloop of the dataflow graph as a separate execution fragment.
 19. Thecomputer-implemented method of claim 1, further including: mapping aplurality of operations from the dataflow graph that are grouped onto aparticular physical compute unit onto resources within the particularphysical compute unit.
 20. A computer-implemented method of transforminga high-level program for mapping onto a reconfigurable data processorwith an array of configurable units, the method including: partitioninga dataflow graph of the high-level program into memory allocations andexecution fragments, wherein the memory allocations represent creationof logical memory spaces in on-processor and/or off-processor memoriesfor data required to implement the dataflow graph, and the executionfragments represent operations on the data; designating the memoryallocations and the execution fragments to virtual configurable units;partitioning the execution fragments into memory fragments and computefragments; assigning the memory fragments and the compute fragments tothe virtual configurable units; allocating the virtual configurableunits to physical configurable units; placing the physical configurableunits onto positions in the array of configurable units and routing dataand control networks between the placed positions; and generating a bitfile with configuration data for the placed positions and the routeddata and control networks, wherein the bit file, when loaded onto aninstance of the array of configurable units, causes the array ofconfigurable units to implement the dataflow graph.
 21. Thecomputer-implemented method of claim 20, further including: allocatingat least one of the virtual configurable units to multiple physicalconfigurable units; placing the multiple physical configurable unitsonto positions in the array of configurable units and routing data andcontrol networks between the placed positions; and generating a bit filewith configuration data for the placed positions and the routed data andcontrol networks, wherein the bit file, when loaded onto an instance ofthe array of configurable units, causes the array of configurable unitsto implement the dataflow graph.
 22. The computer-implemented method ofclaim 21, further including: fusing at least two of the physicalconfigurable units into a single physical configurable unit; placing thesingle physical configurable unit onto positions in the array ofconfigurable units and routing data and control networks between theplaced positions; and generating a bit file with configuration data forthe placed positions and the routed data and control networks, whereinthe bit file, when loaded onto an instance of the array of configurableunits, causes the array of configurable units to implement the dataflowgraph.
 23. A non-transitory computer readable storage medium impressedwith computer program instructions to transform a high-level program formapping onto a reconfigurable data processor with an array ofconfigurable units, the instructions, when executed on a processor,implement a method comprising: partitioning a dataflow graph of thehigh-level program into memory allocations and execution fragments,wherein the memory allocations represent creation of logical memoryspaces in on-processor and/or off-processor memories for data requiredto implement the dataflow graph, and the execution fragments representoperations on the data; designating the memory allocations to virtualmemory units and designating the execution fragments to virtual computeunits; partitioning the execution fragments into memory fragments andcompute fragments; assigning the memory fragments to the virtual memoryunits and assigning the compute fragments to the virtual compute units;allocating the virtual memory units to physical memory units andallocating the virtual compute units to physical compute units; placingthe physical memory units and the physical compute units onto positionsin the array of configurable units and routing data and control networksbetween the placed positions; and generating a bit file withconfiguration data for the placed positions and the routed data andcontrol networks, wherein the bit file, when loaded onto an instance ofthe array of configurable units, causes the array of configurable unitsto implement the dataflow graph.
 24. A system including one or moreprocessors coupled to memory, the memory loaded with computerinstructions to transform a high-level program for mapping onto areconfigurable data processor with an array of configurable units, theinstructions, when executed on the processors, implement actionscomprising: partitioning a dataflow graph of the high-level program intomemory allocations and execution fragments, wherein the memoryallocations represent creation of logical memory spaces in on-processorand/or off-processor memories for data required to implement of thedataflow graph, and the execution fragments represent operations on thedata; designating the memory allocations to virtual memory units anddesignating the execution fragments to virtual compute units;partitioning the execution fragments into memory fragments and computefragments; assigning the memory fragments to the virtual memory unitsand assigning the compute fragments to the virtual compute units;allocating the virtual memory units to physical memory units andallocating the virtual compute units to physical compute units; placingthe physical memory units and the physical compute units onto positionsin the array of configurable units and routing data and control networksbetween the placed positions; and generating a bit file withconfiguration data for the placed positions and the routed data andcontrol networks, wherein the bit file, when loaded onto an instance ofthe array of configurable units, causes the array of configurable unitsto implement the dataflow graph.